Digital video compression systems have been used with great success for both stationary (JPEG) and moving (MDEG) images, and have found widespread acceptance. Discrete Cosine Transform (DCT) based compression systems such as MPEG-2 have essentially solved the transmission bandwidth problem in the present television broadcast environment by compressing more programs into a given band than their analog counterpart.
However, new needs have arisen for further bandwidth reduction. New markets, such as the Internet and HDTV, require a quantum leap in quality-to-bandwidth ratios. For that purpose, multi-layer compression/transmission schemes have been proposed. These include for example, the arrangements disclosed in the following U.S. Pat. Nos. 5,349,383; 5,408,270; 5,418,571; 5,742,343; 5,852,565; 5,988,863; 6,043,846; 6,052,416; 6,173,013; 6,195,390; and 6,229,850. Each of said patents is hereby incorporated herein by reference, each in its entirety. Typically, such multi-layer (or “multi-path” or “multi-channel” arrangements separate the input signal spectrum into a base layer and one or more enhancement layers. The now-common “layer” terminology reflects the typical digital bitstream environment in which signals that would be carried in separate “paths” or “channels” in analog environments and some digital embodiments are instead carried in the same digital bitstream. A problem with multi-layer processes is that the bit rate or bandwidth of the enhancement layer tends to be too high to justify the modest reduction in bit rate or bandwidth achieved and the increase in the complexity of the encoder and decoder.
The use of non-linear enhancement techniques for improving the apparent resolution of a video display is well known. Prior art disclosing such techniques includes a 1951 journal article: “A New Technique for Improving the Sharpness of Television Pictures,” by Goldmark et al, Proc. of the I.R.E., October 1951, pp. 1314-1322; and a number of United States patents, including, for example: U.S. Pat. Nos. 2,740,071; 2,851,522; 4,030,121; 4,504,853; 5,014,119; 5,151,783; 5,237,414; 5,717,789; 5,805,741; 5,844,617; 5,940,141; and 6,108,453. Each of said patents is hereby incorporated herein by reference, each in its entirety. A general principle of such techniques is to generate harmonics of spectral components of a television signal in a controlled fashion in order to extend its high-frequency spectrum. Such techniques may be employed in both the horizontal and vertical picture domain. For example, a typical non-linear enhancement technique detects a transition, extracts the high frequencies defining the rise time of such transition, and performs a controlled harmonic distortion on these frequencies in order to create higher frequencies to be added to the original signal in order to simulate a wider spectrum and provide viewers the illusion of wider bandwidth and greater resolution.
FIGS. 1A and 1B illustrate in the time and frequency domains, respectively, the operation of one type of spectrum expander or spectrum expander function. An input transition of limited bandwidth is enhanced by the addition of an enhancement signal. An output transition is produced which has a significantly increased bandwidth. In the example of FIG. 1B, the input transition has a frequency spectrum extending to F0, whereas the expanded output transition has a frequency spectrum extending to F1. In practice, a ratio of about two is usable for spectral expansion in the frequency domain in the context of the present invention. As indicated below, the ratio is not critical to the invention.
A spectrum expander or spectrum expander function may be implemented in any one of a number of ways, either in the analog or the digital domain. The manner in which the spectral expander is implemented is not critical to the invention. As a first example, the expander or expander function may be implemented as shown in FIG. 1C. In this arrangement, a first differentiator or differentiator function 4 is connected to an input 2 and differentiates the incoming transition (waveform A of FIG. 1E) and puts out the differential (waveform B of FIG. 1E) on a path 6. A full wave rectifier or rectifier function 8 removes the sign of the differential and puts out an absolute value enhancement signal over a path 10 to a multiplier or multiplier function 12. At the same time, a second first differentiator or differentiator function 14 receives the differentiated signal put out on the path 6 from the first differentiator or differentiator function 4. A twice-differentiated signal is then put out on a path 16, e.g., to an amplifier-limiter circuit or function 18 which amplifies and limits positive and negative excursions of the double-differentiated signal and puts out a multiplier-gating signal (waveform C of FIG. 1E) on a path 20 to the multiplier or multiplier function 12. The resultant signal put out from the multiplier or multiplier function 12 on the path 22 is a signal transition having a sharply shortened duration (waveform D of FIG. 1E). FIG. 1D illustrates an alternate arrangement for the spectrum expander or expander function in which the rectifier or rectifier function 8 is placed in the path between the upstream differentiator or differentiator function 4 and the downstream differentiator or differentiator function 14. These illustrations of FIGS. 1C and 1D are applicable to horizontal domain processing and are readily extrapolated to processing in the vertical domain. The techniques described in the present inventor's prior U.S. Pat. No. 4,030,121 are examples of these two approaches of FIG. 1.
Other implementations of the spectrum expander circuit or function may employ, for example, gating techniques as functionally illustrated by the graphs of FIG. 2. The original transition (graph A of FIG. 2) is processed into a second differentiation signal (graph B of FIG. 2) as used in aperture correction for example. As shown in graph C of FIG. 2, when the differentiation signal is combined with the original transition, large preshoot (PS) and overshoot (OS) components are added to the transition and the enhanced transition is too long.
The gating approach takes the aperture correction waveform of graph B of FIG. 2 and develops a gating signal as shown in graph D of FIG. 2. The graph signal D is positioned to be centered at the zero crossing of graph B, and gating of graph B by graph D results in an enhancement waveform, graph E of FIG. 2, which does not have preshoot or overshoot. This signal is then combined with the original transition to produce an enhanced transition graph F of FIG. 2. The gating approach illustrated in FIG. 2 is particularly efficient for spectrum expansion in the vertical domain.
Another approach employs a time delay. A second differentiation of the incoming transition is obtained as shown in FIG. 2B. This signal is graphed as graph A of FIG. 3. This signal is then delayed by a predetermined delay D in a delay line or function. An undelayed component, graph A of FIG. 3 is then compared with a delayed component in an amplitude comparator circuit or function, and a lesser-valued signal is selected and put out from the comparator or comparator function, as graphed in graph B of FIG. 3. The process is repeated for the higher-valued signal as shown in graphs C and D of FIG. 3. Waveform B is delay matched and then compared with waveform D in a comparator circuit. The output of the comparator is composed of the portions of signals B and D having the lesser absolute value of graph B (delayed) and graph D. The resultant is a spectrum expansion signal and is shown as graph E of FIG. 3. This signal is then combined in proper sign, phase and amplitude with the original transition to obtain a resultant having a shortened transition duration.
Another approach is the one proposed by Goldmark et al. in the above-cited journal article. While these techniques build upon a second differentiation of the incoming transition waveform, (a signal necessarily limited by the bandwidth of the incoming signal), the resultant is an enhancement signal which is shorter in time than the original and which has frequency components which are higher than the original transition. Although these techniques are herein described for applications in the horizontal domain (for reasons of simplicity) they can easily be applied in the vertical domain in order to reduce as well vertical domain transitions rise times and give to the image the appearance of a high vertical resolution. In U.S. Pat. No. 5,940,141, a form of vertical bandwidth expansion is utilized.
Analog and digital techniques may be used for bandwidth expansion, although digital approaches are now more practical.
Although such non-linear enhancement techniques have been very successful for video displays, they have not been employed for image processing within systems. The reason is that synthetic high frequencies can only approximate wider bandwidth, and there are many cases when the approach fails. For example, shortcomings of non-linear enhancement include:                Small details, with an entire spectrum above the upper limit of the original signal, have disappeared and cannot be recreated.        Ringing artifacts (e.g. pre-shoots and overshoots) should not be enhanced, and sometimes are.        The amplitudes of the synthetic high frequencies do not necessarily track perfectly the amplitude of the transitions.        Synthetically generated high frequency components most closely match the naturally occurring high frequency components when the non-linear enhancement is optimized for a particular transition rise time. However, transition rise times vary, resulting from, for example, different sources, background purposely out of focus while foreground is in focus, etc., and matching is not necessarily perfect in all cases.        Multiple, repeated transitions (parallel lines) are not very common, but, when present, are not well processed by synthetic high-frequency generators.        
For all these reasons, and others, synthetic high frequencies rarely duplicate real high frequencies, but only approximate them. This approximation, however, can be very good, and the difference between synthetic and real high frequencies is large only from time to time and only in certain picture conditions.